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Abstract. If 



A{x) := ^Cr,-Cx (C>0) 



denotes the error term in the classical Rankin-Selberg problem, then we obtain a 
non-trivial upper bound for the mean square of A(a; + ?7) — A(a;) for a certain range 
oiU = U{X). In particular, under the Lindelof hypothesis for it is shown that 



/ (A{x + U)-A{x)] dx <e js:9/7+e^8/7^ 



Jx 



while under the Lindelof hypothesis for the Rankin-Selberg zeta- function the integral 
is bounded by X^^'^U^^^ . An analogous result for the discrete second moment of 
A{x + U) - A{x) also holds. 



1. Introduction and statement of results 

The classical Rankin-Selberg problem consists of the estimation of the error 
term function 



where the notation is as follows. Let f{z) be a holomorphic cusp form of weight 
K with respect to the full modular group S'L(2, Z), so that 
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when ^mz > and lirnQm z^oo = (see e.g., R.A. Rankin [17] for basic 
notions). We denote by a{n) the n-th Fourier coefficient of (p{z) and suppose that 
<f{z) is a normalized eigenfunction for the Hecke operators T{n), that is, a(l) = 1 
and T{n)ip — a{n)(p for every n e N. The classical example is a(n) = ^{n), when 
K, = 12). This is the Ramanujan r- function defined by 



X 



{{l-x){l-x^){l-x^)---}^^ {\x\<l) 



n=l 



The constant C ( > 0) in (1.1) may be written down explicitly (see e.g., [12]), and 
Cn is the convolution function defined by 



(1.2) 



m 



2(k-1) 



a 



(4) 



This is a multiplicative arithmetic function, namely Cmn = CmCn when (m, n) 
since a{n) is multiplicative. The classical Rankin-Selbcrg bound of 1939 is 



(1.3) 



A(x) = 0(a;^/^), 



hitherto unimproved. In fact, this bound is one of the longest standing unimproved 
bounds of Analytic number theory. In their works, done independently, R.A. 
Rankin [16] derives (1.3) from a general result of E. Landau [15], while A. Selberg 
[19] states the result with no proof. Note that, by the Mobius inversion formula, 
(1.2) is equivalent to 

\a{n)fn^~'^ = ^ ii{d)cn/d'^. 

dp \ n 

Therefore using (1.1), (1-3) and partial summation we obtain 
^ |a(n)|2 = Dx« + 0(a;"-2/5) (£> > 0), 

and conversely the above formula yields (1.1) with (1.3). 

Although it seems very difficult at present to improve the bound in (1.3), re- 
cently there have been some results on the Rankin- Selberg problem (see the au- 
thor's works [5]-[8]), in particular on mean square estimates. Namely, let as usual 
//(cr) denote the Lindelof function 

(1.4) := limsup (a G R). 

t^oo log t 
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Then we have (see [6], [7]; the exponent of ^ was misprinted as 2/(5 — 

(1.5) / /^\x)dx «, /? = I 

Jo 5-4/^(2) 

Here and later s denotes positive constants which may be arbitrarily small, but are 
not necessarily the same at each occurrence, while <Se means that the <S-constant 
depends on e. Note that with the sharpest known result (see M.N. Huxley [2]) 
fii^) ^ 32/205 we obtain /3 = 410/897 = 0.4457079 .... The limit of (1.5) is the 
value f3 = 2/5 if the Lindelof hypothesis for ((s) (that /u(|) = 0) is true. 

In this work we are interested in mean square bounds for A{x + U) — A(a;) in the 
range 1 U ^ X , especially when U is "short", namely when U = o{x) {x 00). 
First of all note that, since c„ <^e , by (1.1) we have 

A(a; + U)- A{x) = ^ Cn-CU 

x<n^x+U 

(l.Dj 

<e ~ 

Although this bound may be considered as "trivial" , there does not exist an ana- 
lytic proof of it yet. Hence using (1.5) and (1.6) we have 

(1.7) / (A{x + U)-A{x)) dx^emm{X^+^'^+',X^+^U^) (1< C/ ^ X). 

One can call then (1.7) the "trivial bound" for the mean square oi A(x+U)—A{x), 
and we seek a non-trivial bound, namely a bound which is (at least in certain ranges 
oiU = U{X)) sharper than (1.7). 

Recently there has been work on the analogue of this problem for some related 
divisor problems. Let Ak{x) denote the error term in the asymptotic formula for 
the summatory function of dk{n), generated by C'^{s) (/c e N). Then in particular 

A2{x) = ^ d{n) - a; (log a; + 27 - l) (d2{n) = d{n) = ^) 

n^x 5\n 

is the error term in the classical Dirichlet divisor problem and 7 = — r'(l) = 
0.5772157 ... is Euler's constant. The author [9] proved that, for 



1< [/ = U{X) ^ \\/X, C3 = Stt-^ 
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and computable constants Cj, we have 

r.2X 

(1.8) Jx 



/ (A2(a; + t/)-A2(x)j da; = TO J] c,- log-' (^j 



Thus for = U{X) ^ X^/^"^ it is seen that (1.8) is a true asymptotic 

formula. 

A result analogous to (1.8) holds if A2{x+U)—A2{x) is replaced by the function 
E{x + U) — E{x), with different constants Cj, where 

E{T) := ^ U\ + it)? dt - T(log £ + 27 - l) 

is the error term in the mean square formula for \C{\ + it)\. For an extensive 
account on E{T) see e.g., F.V. Atkinson's classical work [1], and the author's 
monographs [3], [4]. 

In the general case, when k > 2, the above problem becomes more difficult. In 
[10] we obtained mean square estimates for Ak{x + U) — Ak{x). To formulate the 
results, first we define a{k) as a number satisfying | ^ a{k) < 1, for which 



Jo 



holds for a fixed integer k ^ 2. From zeta-function theory (see [3], and in particular 
Section 7.9 of E.G. Titchmarsh [22]) it is known that such a number exists for any 
given /c e N, but it is not uniquely defined, as one has 

I |C(f7 + it)|2'=dt «, Ti+^ {(j{k)^a<l). 
Jo 

From Chapter 8 of [3] it follows that one has a{2) = |, a(3) = a"(4) — |, a(5) ^ 
9/20 (see W. Zhang [23]) etc., but it is not easy to write down (the best known 
value of) a"(/c) explicitly as a function of k. Note that the Lindelof hypothesis that 
/i(|) = is equivalent to the fact that a{k) = | (V/c e N). Then the result of [10] 
states: Let A; ^ 3 be a fixed integer. If a{k) = |, then 
(1.9) 

i'2X 2 / \ 

/ (^Akix + U)- Akix)j dx X^+'U^/^ (^X'^U = U{X) ^ X^'^j . 
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If I < cr(A;) < 1, and 9{k) is any constant satisfying 2cr(A;) — 1 < d{k) < 1, then 
there exists Si = Si (k) > such that 

It is clear that, if the Lindelof hypothesis is true for C{s), then (1.9) holds for all 
natural numbers k ^ 2. 

Recently the author and J. Wu [11] obtained a new upper bound for 
T^h^H ^kiN, h) for 1 ^ H ^ N, k eN, k ^ 3, where Afc(A^, h) is the (expected) 
error term in the asymptotic formula for J2N<n^2N dk{n)dk{n + h). 

Now we state our results on the mean square of A{x+U) — A{x) as the following 
THEOREM 1. If 11 = /i(|) is defined by (1.4) then, forl^U = U{X) ^ X, 

(1.10) / (A{x+U)-A{x)) da; «, X(9+12M)/(7+4M)+e^8/(7+4M)_ 

If 

(1.11) z{\+it) m + if 

holds, which is the Lindelof hypothesis for the Rankin-Selberg zeta-function, then 
the above integral is bounded by X^^^lf^^^. 

Corollary 1. The bound in (1.10) improves (1.7) for 

j^(l+4M)/(3+4,i) ^jj ^ j^(16m^-8m+9)/(20-16m)_ 

Corollary 2. If the Lindelof hypothesis for C(s) that fi = A*(^) = is true, 
then (1.10) reduces to 

(1.12) j (^A{x + U)- A{x)j dx x^/^+'U^/\ 

and (1.12) improves (1.7) for X^s ^ t/ ^ X^/^o. 

There also exists a discrete analogue of Theorem 1. This is 

THEOREM 2. If /i = //(|) is defined by (1.4) then, forl^U = U{X) ^ X, 

(1.13) Yl (A(n+C/)-A(n))' «e x(9+12p)/(7+4p)+e^8/(7+4M)_ 
X<n^2X 
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If (1.11) holds, then the above sum is bounded byX^+^U"^/^. 

It seems hard to ascertain what should be the true order of magnitude of the 
function A{x + U) — A{x). From (1.8) it seems plausible that 

(1.14) A2{x + U)- A2{x) <£ x^Vu {x^ ^ U{x) ^ x^/^'') , 

which is a very strong conjecture made by M. Jutila [13], but it is not clear whether 
there is sufficient analogy between A{x + U) — A{x) and A2{x + U) — A2{x) to 
make any predictions about the order of A{x + U) — A{x) from (1.14). 

2. Proof of the Theorems 

There are two natural tools to study A{x). The first is the explicit, truncated 
formula for A (a;), of the Voronoi' type, namely 

A{x) = ^Y1 <^kk-'/'sm (sTTikxy/^ + ^) + 0,(xW=K-i/4)^ 

where the parameter K satisfies 1 <^ K <^ x. The proof of this result can be 
found in [12]. However, the error term is much too large for our present purpose. 
Therefore we resort to the use of another natural tool in the study of A{x). This 
is the Rankin-Selberg zeta-function 

oo 

(2.1) Z{s) := 5]cnn-^ 

n=l 

defined initially for s — a + it,a > 1, and for other values of s by analytic 
continuation. It has a simple pole at s = 1 with residue equal to C (cf. (1-1)), 
and is otherwise regular. For every s e C it satisfies the functional equation 

(2.2) T{s + K- l)T{s)Z{s) = {27t)^'-'^T{k - s)r{l - s)Z{l - s). 

The Rankin-Sclberg zeta-function Z{s) belongs to the Selberg class S of Dirichlet 
series of degree four. For the definition and properties of S see e.g., the seminal 
paper [20] of A. Selberg and the review paper of Kaczorowski-Perelli [14]. 

One also has the decomposition 

oo oc 

(2.3) Z{s) := 5^ cnfi-^ = as) Yl bnfi-' = as)B{s), 

n=l n=l 

say, where B{s) belongs to the class S of of degree three, and moreover the function 
B{s) is holomorphic for 3ftes > 0. This follows from G. Shimura's work [21] (see 
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also A. Sankaranarayanan [18]). The coefficients bn in (2.3) are multiplicative and 
satisfy bn <Se (see [18]). Actually the coefficients bn are bounded by a log-power 
in mean square, but this stronger property is not needed here. 

If we suppose that 

(2.4) / \B{^+it)\''dt <^e X^+' (0^1), 
Jx 

and use the elementary fact (see Chapters 7 and 8 of [3] for the results on the 
moments of |C(^ + ^^)l) that 

p2X 

(2.5) / |C(i + ^^)|^d^ < XlogX, 

Jx 

then from (2.3)-(2.5) and the Cauchy-Schwarz inequality for integrals we obtain 



(2.6) 



/.2X 

/ \Z{^+zt)\dt «, 
Jx 



As B{s) belongs to the Selberg class of degree three, then S(| + it) in (2.5) can 
be written as a sum of two Dirichlet polynomials (e.g., by the reflection principle 
discussed in [3, Chapter 4]), each of length <^ X^/^, plus a manageable error term. 
Thus by the mean value theorem for Dirichlet polynomials (op. cit.) we have 
9 ^ 3/2, and any improvement on the value of 9 would give an improvement of 
(1.3), as shown by the author in [6], [7]. 

To prove (1.10), we start from (2.1) and Perron's inversion formula (see e.g., 
the Appendix of [3]) to obtain 

1 pl+S+iT s 

(2.7) y^cn=— - Z^s) ds + 0,(Xi+^T-^), 

where X ^ 2X, 1 < r < X and T ^ r ^ 2T will be suitably chosen a little 
later. We replace the segment of integration by the contour joining the points 

1 + e — ir, ^ — IT, ^ + ir, 1 + e + ir. 

We encounter the simple pole of Z{s) at s = 1 of and the residue will furnish Cx, 
the main term in (1.1). Hence by the residue theorem (2.7) gives, once with x and 
once with x + U, 

A{x + U) - A{x) = ^ /'^'" (^ + ^y-^' Z(s)ds 



(2.8) 27riyi_,, s 

+ Oe{X^+'T-^) + 0{R{x,t)) 
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where we set 



R{x, t) 



1 

T ./I 



l+£ 



x''\Z{a + iT)\da. 



From (2.6) (with 9 = 3/2) and the convexity of mean values (see e.g., [3, Lemma 
8.3]) we have 

(2.9) / |Z(a + zt)|dt«,T(^-")/2+= (1^(7^1), 

Jt 

and the integral in (2.9) is <^e yi+e for cr ^ 1. It follows that 

j R{x,T)dT ^ J^^ {^j |Z(cT + ir)|drj d(T 



1 



max 



X 



since T ^ X. Note that this holds uniformly in X ^ x ^ 2X. Therefore there 
exists To G [T, 2T] for which 



i?(a;,To) <e x'+^T"' (X ^ a; ^ 2X) 

holds uniformly in x. It is r = Tq that is chosen in (2.7) and T is the basic 
parameter to be determined. Then using 



(a; + Uy - x' 



we obtain from (2.8), since T ^ Tq ^ 2T, 
(2.10) 



U 



{x + vY dv 



A(a; + U)- A(a;) 



1_ ( t 



{x + vy-^ dv 1 Z{s) ds + Oe{X'+^T-') 



l+erp-l-^ 



On squaring (2.10) and integrating, we obtain 
J (^A{x + U) - A(x)y dx 

<e / / / {x + v)^-^+''Z{l+it)dvdt dx + X^+'T-^. 

Jx J-T Jo 



(2.11) 
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Let now il>{x) 0) be a smooth function supported in [X/2, 5X/2], such that 
i;{x) = 1 when X ^ a; ^ 2X and i^^'^^x) <^ X"^ (r = 0,1,2,...). By using 
the Cauchy-Schwarz inequahty for integrals it is seen that the integral on the 
right-hand side of (2.11) does not exceed 

5X/2 pT ^ 2 

dv dx 



U ip{x) / {x + v)-^+''Z{^+it)dt 

Jx/2 Jo J-T 

pU pT pT 

= U / / Z{^+it)Z{^-iy)J dy dt dv 

Jo J-T J-T 



say, where 

J = J{X;v,t,y):= / il:{x){x + v)-^{x + vf^^-^^ dx. 

Jx/2 

Integrating by parts we obtain, since 'i(j{X/2) — il){bX/2) — 0, 

-1 /"^-^/^ / 1 \ 

J = ^ / (a; + vY^'-y^ ( ij'(x) V'(^) ) da;. 

i{t-y) + lJx/2 ' ' x + v^^ ') 

By repeating this process it is seen that each time our integrand will be decreased 
by the factor of order 

\t-y\ + l X 

for |t — 2/1 ^ X^ . Thus if we fix any A > 0, the contribution of\t — y\ ^ X'^ will be 
<S X~-^ if we integrate by parts r = r{e,A) times. For \t — y\ ^ X^ we estimate 
the corresponding contribution to J trivially as 0(1) to obtain that the integral 
on the right-hand side of (2.11) is 



-^eU^ [ [ \Z{^+it)Z{^+iy)\dy dt + 1 

J-T J-T,\t-y\^X^ 

J \Z{l+it)\^ dy^ dt + 1 



Here we used the elementary inequality \ab\ ^ + I^P)) £^nd the bound (cf. 

(2.4) with 9 = 3/2) 

p2X p2X 

/ \Z{^ + it)\^dt^ / \B{^+it)\^\C{^+it)\''dt 
Jx Jx 

p2X 

T^'^CD+s / |5( 1 + it)|2 yi+2M(i)+s_ 

Jx 
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Therefore it is seen that the left-hand side of (2.11) is 

(2.12) <s X%U^Ti+^^'^^^ + X^T-^). 
With the choice 

T = X^/^5+^''(5))[/-2/(i+2M(5)) 

the terms in (2.12) are equahzed. The condition 1 ^ T ^ X is trivial, and (2.12) 
yields (1.10). Note that in proving (1.9) we could use power moments of |C(| +it)\ 
for which there is certainly more information than for the moments of |Z(t)|. This 
reflects the quality of the bounds in (1.9) and (1.10). 

Finally note that if (1.11) holds, which is the Lindelof hypothesis for the 
Rankin-Selberg zeta-function, then obviously 

(2.13) / \Z{l+it)\^dt^eX^+\ 

Jx 

This would replace (2.12) by 

The choice T = Xt/-^/^ yields then 

(2.14) j (^A(a; + t/) - A(a;) j dx <e 

which is non-trivial in the whole range 1 -C [/ ^ X. Clearly for the proof (2.13) 
suffices instead of the stronger (1.11). The bound (2.14) is the analogue of (1.9). 
This completes the proof of Theorem 1. 

To prove (1.13) of Theorem 2 we employ the method developed in [9]. We 
can assume that U and X are natural numbers, for otherwise we shall make an 
admissible error by using trivial estimation. Using (1.1) it is seen that integral in 
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2 

f-m+l-O 



(1.10) is equal to 

/■m-i-i-u / \ 

X^m^2X-l-^'^ \x<n^x+U J 

/■m+1-0 / \ ^ 

= E / E Cr^-CU] dx 

/■m+1-0 

J2 / (^(™ + C/) - A(m))^ dx 



X<m<2X-l 



(A(m + C/) - A(m))^ 
J] (A(n + C/)- A(n))' + Oe(X^C/). 



X<nC2X 



Here in the last step we used (1.6). Since the error term above is absorbed in the 
expression on the right-hand side of (1.13), the proof of Theorem 2 is finished. 
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